Skip to content

feat(storage): automate graph disk cleanup and controlled legacy retirement - #153

Merged
rabii-chaarani merged 72 commits into
mainfrom
codex/automatic-graph-disk-cleanup
Oct 27, 2026
Merged

rabii-chaarani merged 72 commits into
mainfrom
codex/automatic-graph-disk-cleanup

Conversation

@rabii-chaarani

@rabii-chaarani rabii-chaarani commented Oct 20, 2026 •

Copy link
Copy Markdown
Owner

This draft implements automatic graph disk cleanup and controlled legacy retirement.

Supported native platforms are macOS ARM64 and Windows x86_64. Linux and Intel macOS targets, release requirements, installer support and bundled extensions have been removed.

Native CI runs serial test cohorts with complete discovered inventories, bounded Windows groups and artifact production gated on every test group. Storage recovery now completes authenticated pending control transactions before role dispatch. Coordinator and lifecycle fixtures preserve ownership, containment and deadline assertions.

All required CI checks and both native release build-and-smoke jobs passed on 6e1543b.

Successful CI: https://github.com/rabii-chaarani/codebaseGraph/actions/runs/40189203041

Approved Scope A — safe Ladybug COPY loading

Added in 8e47d2b3c74b72b56a0b6b212d1bf939c8088ca7 under CBG-COPY-01 and the recorded user approval.

  • Execute each mutating phase once at its configured pool; remove same-candidate and fallback write retries. Native errors are no longer hidden by existing-object substring matching.
  • Reuse the configured spill bound for final sequential JSON/CSV COPY files and retain early primary-key indexing. No JSON list/glob consolidation or native dependency upgrade.
  • Bound stderr ingestion and assembled context to the existing budget; preserve COPY-query versus explicit-checkpoint failures even for oversized native errors.
  • Test real CLI/isolated helpers, Managed/Direct prior-bundle preservation after real COPY success plus acknowledgement failure, subsequent fresh invocation, PK/index/connector/search correctness and same-source chunked/control parity.

Local macOS ARM64 checks: focused 72/72, final writer/diagnostic/pool-matrix15/15, xtask42/42, doctests3/3, optimized native COPY2/2, format and strict workspace/all-target/all-feature Clippy passed. Fresh GPT-6.1 Sol/Max independent review found no unresolved confirmed source defects after fixes. Execution and limits.

Pending: the full1,344-case serial local workspace run, existing optimized lifecycle/resource cases, and the new commit's native-platform CI (including Windows x86_64). Earlier successful CI above applies only to its named old SHA, not this new commit. No merge/deployment/daemon restart, Scope B automatic reconstruction or Ladybug0.22 adoption is included. Linear synchronization remains pending connector verification.

Approved follow-up — remove workload resource ceilings

Implemented in 2243e590493d917079c12f719624bc89477842a4. Approved R1 amendment and user approval. This supersedes the earlier Scope A statements about preserving worker/Rust memory ceilings and its configured-pool policy.

Removed only the workload budget enforcement and options: worker_memory_mib, rust_memory_mib, their CLI/config/API plumbing, artificial source/partition/artifact admission checks, RSS-triggered termination and application-derived native buffer pool. Native engine defaults remain physical/native policy, not an unlimited-resource guarantee.

Retained spill/chunk/concurrency tuning, bounded diagnostics/protocol/search-format checks, fail-closed COPY without replay, ownership/containment/cancellation and maintenance per-pass safeguards. Individually large valid records can occupy a singleton chunk/run rather than failing for exceeding a batching target.

Local post-removal checks passed: 134 focused cases; 42 settings/API/ownership cases; 2 real CLI COPY/control/reopen/preservation cases; 1 legacy-budget-config case; 1 owned-allocation case; 2 mixed/jumbo merge cases; 1 actual MCP schema case; 1 jumbo native JSON persistence/reopen case; 3 doctests; final format, workspace all-target/all-feature compile and strict Clippy. Independent fresh GPT-6.1 Sol/Max review found no confirmed P0–P3 production defect. Retained cancellation passed after the source commit.

Retained cancellation and unknown-containment cases both passed after removal (1/1 each); the follow-up evidence is recorded in 088415dbcbc2934a9464e1b0d8e968395d80b2fe.

Pending qualification: the new head's full native-platform CI (macOS ARM64 and Windows x86_64). Earlier CI failures/passes belong to their named previous SHA; no broad passing qualification or merge/deployment is inferred. Executed evidence and limits.

Guard producer effects and scoped runs with reference epochs. Keep optional cleanup failures from blocking committed reads, and report post-publication artifact cleanup separately. Experimental collector and validator prototypes remain excluded.
BREAKING CHANGE: database open helpers return GuardedDatabase and reject path forms whose native resource identity cannot be proven. Connections borrow the guarded handle; canonical Windows extended paths retain their namespace.
@rabii-chaarani
rabii-chaarani marked this pull request as ready for review October 25, 2026 23:30
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 25, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-25T23:33:42.499925Z 6e1543b Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6e1543b038

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/storage/maintenance_candidate.rs
Comment thread src/storage/maintenance_legacy_generations_engine.rs Outdated
@rabii-chaarani

Copy link
Copy Markdown
Owner Author

Fixed the cargo clippy failure in 862a8f88d485d5bd257ebd9456c194156fd4ee94.

Root cause: removing resource-budget callers left LadybugWritePhase::trace_label used only by the macOS timing wrapper. Ubuntu compiled the helper but not its caller, so -D warnings failed on dead code. The fix applies the same cfg(target_os = "macos") to the helper implementation. No warning suppression, runtime behavior, resource ceiling, dependency or constant changed.

Failed job: https://github.com/rabii-chaarani/codebaseGraph/actions/runs/40329652884/job/121595748650

Verified locally on macOS ARM64: strict workspace/all-target/all-feature Clippy, formatting/diff checks, and all 12 affected phase tests pass. New Linux CI remains the platform verification gate.

@rabii-chaarani

Copy link
Copy Markdown
Owner Author

Approved CI-only maintenance fixes

Implemented COD-57, COD-58 and COD-59 only in d6dd34493e9da6f80847240bafcc92dbcaed2c38. Exact user approval binds spec revision0.10 1a9db7b558b60248888408daa6268a3b45f2f0df; runtime packaging, other platform maintenance, merge/deploy and restarts are excluded.

  • Watch help now states the unchanged parallel default.
  • COPY trace fixture uses the actual production path renderer and a spaced filename; canonical file identity and exact4-byte oracle retained. Production parser/COPY behavior unchanged.
  • Epoch observer recognizes fs2 raw native contention in addition to WouldBlock; original5s deadline,10ms interval and full birth/epoch/integrity assertions retained. Added Windows owned contention/release and missing-lock fixture tests.

Local macOS ARM64: 44 selected tests passed (42unit +1missing-lock +1real MCP first-creation/followup publication case), normal CLI build, formatting, strict workspace/all-target/all-feature Clippy, focused independent review with no confirmed findings. Actual last full fixture snapshot MCP case passed69.46s. Windows-only contention fixture has not been run locally.

Durable execution/review evidence. New exact-head CI run 40426019260 is in progress, not passing qualification yet. Linear issues are In Review, not Done. No production constants changed; test-only synchronization waits are described in evidence. No merge, deployment or service restart performed.

@rabii-chaarani

Copy link
Copy Markdown
Owner Author

Approved CI-only fixes: native verification passed

CI run40426019260 attempt1 is success,22/22 jobs, bound to head d6dd34493e9da6f80847240bafcc92dbcaed2c38. Both native artifact build-and-smoke jobs and required aggregate executed/passed.

Actual checkout was synthetic merge af71103436086bff1189e67e0f5f03464e977bd4 into ca115690b226384164cf364458a0e3255b8908d3; comparison to the delivered head changes only CHANGELOG.md. Durable exact-source/job/binary/case evidence and named testcase PASS verification.

Confirmed actual executions: watch-help on Mac+Windows, COPY metadata on Windows, real followup publication-birth/epoch on Windows, new Windows owned-contention/release regression, missing-lock propagation. No job-green-only or inventory-only inference.

COD-57, COD-58, COD-59 are Done for the approved CI-only scope. Other Maintenance work, COD-51 remaining approval gate, COD-61 clean desktop/minimum-OS/installer/source-install qualification and aggregate milestone remain open. This does not authorize merge/deploy/restart or implement packaging/discovery/cursor/lock-mode policies. No further product source/constants changed.

@rabii-chaarani
rabii-chaarani merged commit 75a99c8 into main Oct 27, 2026
22 checks passed
This was referenced Oct 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant